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ABSTRACT , \ 

The new HIE (Nationsll. Institute of £du<^ation) 
prolects. Compensatory Education Study (Hill et al.,- 19>€, 1977a,, 
1977bJ and the Vocational Education Study /(David et .al.^^ t978) , along 
with an eaxlier Safe Schools Study, are feeing^ consi'deted in the 
evaluation cbmaunity* as possible new paradigis for research. Eesults 
of previous evaluation studies reveal several problesss, aaong which 
were overly technical findings, reports arriving tO|C early or too , 
late, poorly designed evaluations, data regijest^d only -^o bolster 
previous positions, etc., Characteristics for the new evaluations 
involve direct routes to Congress, considerable tiae allocation, and 
absence of constraints in the legislation on the design, oeasures, or 
analysis. Despite the improvements in the approach to evaluation, 
several problems are still encountered: Congress often ii;idirectly 
affects thjer evaluation paradigm when committee memberships change, 
resulting in a lack of, knowledge concerning the design of a project 
and, its purpose; (2) -although an agency chosen tp h^indle mandated 
studies needs to be able to guickly hire and fire perscnriel to \ 
procure talent for short periods and to guickly contract work, staffs 
to manage projects must be accommodated within agency ceilings, job 
expenditures must be negotiated with Congress, and inhibiting civil 
service procedure must be followed; (3) difficulty is encountered by 
evaluation teams who must effect communication with dii^erse groqps 
within an organization; and (^l) mandated studies cannct conflict. 
Though there is little research on mandated studies, it is concluded' 
that evaluati^on cannot evolve *^guickly and adaptations must be made 
whila the evaluation is in progress. (VB) 
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ouceo Exactly as received prom 

THE PERSON 0*^ ORGANIZATION ORIGIN- 



fJl^5l '^^''^^^^ VIEW OR OPINIONS , T^4«-^114« r^^*.*.^ 

STATED cx> NOT NECESSARILY REPRE. LoiS'-ellin Datta 

"^^S:^^^^^^^^^ . National Institute of Education* . ^^o^^"™,^-^ «^ 

' ^ * • , * USeBS OF THE ERIC SYSTEM" 

O Between 1972 and 1976,* Congress asked the National Institute of 

\£\ * Education (NIjE) to conduct two evaluation reseatch projects. These* 

bo are the Compensatory Education Study (Hill et aU, 1976^, 1977a, 1977b) 

' ^i^d the Vocational Education Study (David et-al,, 1978). The Safe 

Schbols Study (Boesel et al., .1978) was mandated as the responsibility 
of the Secretary of Health, Education and Welfare and ^ssi^ned first to 
the National Center for Educational Statistics; in 1975, the study was 
l7f transferred to Institute for" design and management. The purpose 

of this paper is to share soroetihing of what has been learned about this 
form of evaluation in the past few years, a report something akin to an 
early. description of what may (or may nofc) be a new species, 

/ • ' 

Congressionally mandated studies ar/e, nothing new. For many years, 
Congress has asked for research from diVerse agencies, three of which 
/ ' report ex'clusively to^Congress (Beckman, 1976). Since 1946, 'the 
. Congressional Research Service of the Library of Congress (CRS) has 
answered Congressional requests for knowledge synthesejs and analyses. 
*The General Accounting Office (CAO). provides Congress with substantive 
audits of program administration, management and effectiveness as well 
as reports on fiscal accountability* GAO's stature as a tough^inde,d, 
independent agency is bolstered by appointment of its director for a 
16 year term.- While many GAO* studies in. educatioii have relied on 
^ secondary sources such as annual state reports , GAO increasingly is 

collecting H.ts first-hand information and verifying the evaluation 
reports of others through such devices as secondary analyses and re^ 
interviewing participants (Marvin et al., 1977)., The CRS and GAO -were 
joined in 1974 by the Congressional Budget Office. In a very short time', 
the new Office has proven to have formidable, capabilities* in educational 
pplicy studies as well as analyses in related fields. 

\ ' " Opinions ar6 the author's. Endorsement by the National Institute of 
Education should not be inferred. My thanks to David Boesel, Henry 
David; Jim Harvey and Bob Harris for comments on an earlier draft. ^ 
Papea: prepared for the American Educational Research Association mating i 
Toronto,' March 27-3-1, 1978. References on request,' 




f Ih addition to these three agencies reporting only t>o Congress, 

.the Legislative Branch .has turned often to the Executive Branch agencies 
and to independent groups sifch as the National Academy of Science for * 
evaluative and policy research. In almost all educational legislation, 

'Congress has directed the Executive Branch to collect data for Congressional 
use. While many Congressional directives in educational evaluation are 
intended to-^improve program management, the jxrecedent for expecting 
other information more attuned to Congressional needs goes back to the 
Act authorizing a Department of Ediication in 1874 CCronbach and Suppes, 
1^68).- As an example in>ocational education legislation, the 1963 Act 
required appointment .of a 1966 national commission to report on the 
implementation of the 1963 law prior. to. the re-authorization expected 
in 1968 (Datta, 1978). * * • 

Despite a very long list of prior Congressionally mandated studies, 
the new NIE studies are attracftin'g- some attention in the evaluation 
community as a possible new paradigm for researclj. The interest is 
understandable. Not only has the Compensatory Education Study already 
featured prominently in re-authorization testimony (Hearings^ 1978), but 
in a young field, models are a useful way of organizing theo'ry and 
prafctice. Already in evaluation research, much interest has center.ed 
on the distinction between summative and formative evaluation, on planned 
variations, on the local problem solving model, on adversary evaluation, 
and on ap,^roaches such as CIP and MAUT* Theoretical discussion and early 
enthusiasm, however, have somewhat outpaced field experiences with these 
models. Where models are applied in situations later experience shows 
were inappropriate, premature pessimism about the mo.del and evaluation 
can replace premature optimism. Planned variation is a case in point; 
despite the effort ^to learn from the Follow Through experience (Kivlin ^ 
' and Timpane, 1976), there ha^e been no planned variation* evaluations 
begun since 1972. 

Habit at * x 

Before turning to the features of this possible new model whxch 
may help predict where it. may be appropriately used, ^ bit abo^t its 
natural habitat may be helpful. All three* studies appeared at a time 
when early rumblings about the failure to act on the findings of 
evaluation research (see, for example, Suchman, 1966, and Who ley et al., 
1968) ^reached earthquake proportions. These alleged failures have 
led some to the» safer ground of not expecting mudh direct guidance 
for policy-pakers from social research (for example, Cohen and Caret, 
1975), OthWl have examined the political milieu in which evaluation 
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research has occurred, turning -to s.tudies of thjs utilizati(fe processes 
under -varied conditions (for example, Weiss, 1972 and Havelock, 1975). 
Still others 1\a3je tri^d harder with various technical remedies for 
increasing the utility and th^ utilization of evaluations (for example, 
-^Rivlin and Timpane, 1976; peneral Accounting Office, 1977). A robust 
new discipline is emerging from these efforts^with models, theories and 
projects of its own: the field of, knowledge production and utilization. 

Leaving aside the question of ,whether non-utili-zation is so 
universally the case (and I do not think i\ is), a number of problems 
have been fingered as the culprit. The charge's include: 

0 findings have been repotted in" too technical manner 

0 the content and focus of th^ studies are unresponsive 
to the 'needs of policy-makers: the wrong questions are 
being asked - ■ • . 

0 release of reports and times of decision are-out of 
synchrony; 'Reports are too early or too late ' 

o the evaluations have been poorly designed and the 
/ measures have been i'napprppriate or* insensitive 

0 evaluators have not had enough- time, money or authority i 
to ciarry out the studies properly; the techniques are 
adequate but the conditions for using these techniques 
have been wanting * * * . 

0 report quality^ has been attetttuated by insufficient 
negotiations with too few layers of power*. 
* " * * 

^ ^ 0 report quali-ty has bien attentuated by too much negotiation 

. with too many layers of power 

0 the evaluation paradigms have be^n appropriate for go/no go 
'decisions i:ather than the more modest,, ipai^inal changes in > 
characteristics of our system 

0 policy-makers * don't, want the data, except to bolster 
selectively their pre-conceived and ?.mmovabl^ positions 
on issues , - , 

The* new-Congfessionally mandated studies "differ from other ^ • 
requests in ways related to these diagnoses:.^' While otlier studies »may 
share some of their charactei^istics, thfe new studies have them, all, 
albeit in .varying degrees. , • - 



o' .the timing of th^ studies* is ^specified in the law 
' tro be compatible with the needs of Congress 

0 the^ studies requite pricfr approval by Congress" of a 
plan o5 research, in which the mandate in the legi^-r 

^ lation undergoes a firs.t transformation to a series 
of research questions / 



0 research project designs an4 measures- evolve through 
^ negotiations with Congress r The negotiations and dis^ . 
cussions continue throughout the course o£ the ifesearch 
in something of an archifect-'client relationship, 
with the client observing the building taking form as . 
the Vesearch findings come in 

o the client is unequivocally Congress. While many other 
groups m^ay be asked for^advice in the course of the study. 
Congress is the .client the NIE staff and consultahts are 
' serving * & 

0 the studies have dir^t totites to Congress without prior 
Reviews by any external group or meniBer of the Executive ' 
Branch * . 

o funds are provided directly in the legislation for these 
stud^ies so they do not- have to coimete for money with other 
Executive Branch interests ^ ^ 

o the studies have considerable timf ^^Cfrom one i?eauthori2a\ 
tion to the next, a three to five year period), and thus 
have a running sj:art ou adequate measurement, design, 
field work and analysis 

b .there are no constraint^ in the legislation on the design, 
measures or analys is . Technically , only the state of the 
art and and/ability. of the researchers should limit the 
quality of the'work < 

S several related studies together form a plan of evalua-* * 
^ tive research. A mix of I'arge and of small studies, jof 
'surveys, policy analyses, experiments, case studies, cost 
assessments,, process and outcome evaluations can converge 
on questions of Congressional, interest . The product is 
\ information answering these questions, brought togethetr 
from a variety of -sources. , 



b the studies are given specific authority t^ experiment 
. with approaches to eligibility and allocation formulae 
and other requiremeftts of the law which are otherwise 
^ prohibited \ t 3 

!♦ * ■ . *• 

0 Congress specifies fairly precisely in the legislatid.n 
the questions it wants answered. Meetings with the 
legislators and their staffs, reports of discussions 
* during t|\e hearings, and examination of the testimony 

of witnesses are also available. These help- the eval- 
uators understand "the purposes of the law and why the 
questions Vere important enough to re\«Lre a specially 
mandated study. •> * " • . * 

This list of evaluation characteristics perhaps comes as close to 
heaven _ as ^many evaluators have dreamed". ; There are, it should 'be noted, . 
variation. Both the Safe Schools Study-and the Compensatory Education 
Study neefed extensions of time. There were no formal requirements 
in th% Safe Schools Study for review of plans, although the study was 
discussed informally and at length with Congressional staff. Safe 
Schools Study findings were reported to <;ongress. by .the Office of the 
Secretaryof Health, Education and Welfare after review, which in this 
instance was scrupulously- apolitical though somewhat slow. Funds for * 
the Safe Schools Study were not provided by Congress and hid to be 
scraped up; the law .did require a certain data collection procedure 
and timeJ-T-ne which proved unworkable; and the study was not given 
authority for experimentation. -Th^' Vocational Education Study", while 
blessed with "all of the features, "has not to date involved the close 
communication with Congress ^ that may have been a pivotal aspect of the 
Compensatory Educa'tion Study. •. ' 

The studies also vaiy in their sfage of, development .' The Compen- 
satory-Education Study is. near completion, althglGgh data ^re still' being 
collected m one project and analyses sugges^d/by Congressional interest 
in earlier reports are still ongoing.. " The re-f thbrizatioa hearings on 
the Elementary an'd Se^on<fary Education Act begin almost a year ago. 
They are still in progress, and the. eventual iipact of the research on ' 
legislation is unsure. The immediate impact is observable in the quality 
of the discussions anti ^testimony in the hearings,- and in the extent to 
which- the NIE study« findings are citdd as credible in informing discussion 
of the issues. The. Safe Schools Study has just released its report. 
No policy effects are visible. as yet, but the findings have received 
considerable attention in. the preSs. And the Vocational Education^ Study 
is just entering the Second year of a four year effort. ' * 



6- 



" Conments* will have to be based .then, on confounding this possible 
new species with its administration by^a single agency, with that agency's 
turbulent history, with the characteristics of the handful of people who 
have participated in the studies in the Institute and in Congress^and 
*.^^with the nature *of the problems, examined and the constituencies involved. 
With three incomplete specimens, conclusions about the n^w' Congress ionally 
mandated study as a paradigm for improving the utility and utilization 
of research are premature', but Speculation is irresistable. 

Some Boundary Conditions ^ . % . , 

'A Good Friand but a Complex Master ; Glennan and Berryman ' (19.7,8) have 
suggested that the Office of Education focus its evaluation moneJ« cm_ 
the USOE's big budget programs such as student loans, vpcatiotial ' " 
education, and compensatory education, and that, all its* studies adopt 
the Congressionally negotiated features of the Compensatory Education 
study. 

* . 

One boundary condition .on the use of these features is that the 
many oth-er demands on tl^e time of Congressional staff place a very low 
upper limit on how many negotiated studies could be \iaxSSt^, As an 
example, between February 15th and May '15tjk the Senate subcommittee 
responsible for education legislation mua^'eport out of Committees 
■ fi ft ee n major pieces of educational legifelWion which are up >for re- 
rfeauthoriz'ation in 1978, in addition to miaiiy pieces Sf new legislation 
committee members want considered. -The ftouse counterpart sub-committee 
responsible for education legislation has to deal with 'all of 'these 
^ reauthorizations plus, its own slate of new legis-lation. While authori- 
zations are- on a three to five year cycle,' the Houaie- and , Senate appro- 
priations committees operate on a yearly cycle, having to review each - 
year the funds to be allocated for every single piece- of educational 
legislation. ' The staffs of the coninitteea are dealing on a'per person 
basis with revels of funds which make the rule-of-thurab "of evaluation 
management ($500,000 for orre professiona.l) seem indolent luxury., - 
■While all Congressional staffs and the I'egislators themselves have been 
monumentally generous with their time on the three NIE dtudie'g, tfheir 
time is severely limited. It musfbe'used with a'-grueling efficiency 
that probably would rule out many Evaluations with involvemetrtri^e . 
intense as that given to the Compensatory Edupation study> 

* Another aspect of Congress affecting mandated or negoti^teci '§tufiies 
as a model is illustrated by two instances. Between authorization oM^.^ ' 
the Vocational Education Study in 1976 and completion of the. study plan 
in December 19774 Congressional Committee membership changed. It will 
.change at least one more before the study is* completed] In the f^rst 
change, one member who most wanted one of .the four sub-studies left the' 
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Commrttee. The members of Congress to whom the issues were highljr 
important may retain interest if they remain in Congresjs, but other ^ 
matters in their new Committee assignments may become more pressing. 
The study design can not assume that four years latej:, there will be 
^many people on the subcommittees who remember why the study was so 
important to begin with pr who will welcome the findings. 

There are, to be sure, continuities. The Chairmen of a Sub- 
committee and of a full Committee are by definition senior members 
of Congress who exercise great leadership within their committees. 
They are likely to continue to serve in Congress, along with their 
staffs long after the study is completed. Representative. Carl 
Perkins, for example, has chaired the House Committee concerned with 
education for over 15 years. Also, the study questions can be 
formulated to anticipate the durable issues, avoiding more faddish 
concerns. As examples, j^e consecfuences o£ changing eligibility or 
allocation formulaue are durable issues.- Studying how rapidly the 
energy industry expands as a result of vocational education energy 
occupation curriculum development might be of shorter t^rm value. 

To a great extent, responsibility for selecting durable issues 
rests in the Congressional committees writing the legislation man-- 
dating the study ^ Nofc all members of C^sngress seem to have a firm 
grasp on research or how to write legislation permitting an achievable 
study. Some National Academy of Science stiff, for example, recall 
with awe one Congress ionally mandated study which^ required a compre- 
hensive report on all aspects of all consequences of all uses of all 
defoliants' in Vietnam, to be complete^ in six months for a total 
expenditure of $15,000,000, During the planning process, however, 
there is ofjportunity to see how well different ways of formulating 
research questions, may meet the needs underlying the language in the 
legislation and. to select the issues of a more durable and do'-able 
nature, 

* 

.A second instance is the many ways insjwhich the evaluator is 
reminded of the diversitjr of Congress. In the Vocational Ediication 
study,* for example, there is diversity even within the House and the 
Senate conmittees responsible for educational legislation on such » 
matters as: * ' f * 

I ^ ' » 

0 the primary purposes of Federal .support for vocational educatio 

0 the priorities within the- legislation of different goals 
such as ^ssuring post-program/ Employment in training- 
related occupations versus serving most economically 
depressed areas or assuring ^e^fairness in all aspects 
of the program ' ^ . - 



d the purpose of the NIE study as an independent report 
to Congresa oil the*responsive'i^ss of the USOE-admin- 
"astered program to the new legislg.tion or as a tnore 
futures oriented ^examination, of the relationships between 
Inderal vocational education .and other education, ^nd 
between vocational education and other forms of economic 
development policies , ' '^^^ 

Congressional diversity also has implicationa for the study on a more 
mundane\ level. As an example, in the draft legislation, research 
to inform three questions was tOs receive an authorization pf $2 
million annually. During the House-Senate conference when the worth- 
whileness -of the study 'was challenged, a fourth large question was 
ajided bul: the budgit was cut in half. While $1 million annually 
for four years is a. substantial sum, the consequences of increasing 
the assignment by 33% and decreasing the funds by 50%, has challenged/ 
staff ingenuity. And we are aware that although , Congress has backed/] 
up 'its mandated studies, the studies have to walk a line between * 
outraging large, well'^organized^onstituencies that rarely welcome 
evaluations and assurance that the research will be thorough, objec&ve 
and "unbought". ' | 

Orphans in Bureaucracyville : As one researcher observed, 

"Ta handle Congressionally mandated studies, an agency needs 
to operate like a grivate research firm; able quickly to hii^ 
and if need*be, to fire, personnel; to bring the talent neeafed • 
quickly on board for short-term assignments; and to arrange/' 
r^idly for contracted work. A kind of orphan status and / 
the glacial procedures for doing anytfhing combine to almost 
• sink the studies". ^ . 7 

The Congressionally mandated studies, while bringing much prestige to 
agencies, are-'s^ill frozen money which can not be directed to the 
priorities of agency administrators. Two of the studies have befen 
provided with funds from USOE programs, $5 million annually 'for three 
years for the compensatory education projects and i?l million annually 
for four ye^rS for the vocational education study. Although the • 
direct costs of these two studies do not bite into NIE appropriations, 
agency administrators ca|i not use these monies for projects of special 
interest to them. And tfie staff to manage these projects must be 
accommodated'within the agency ceiling, subject tq whatever r^organiz^a- 
tion, freezes or downgrading is currently affecting the agency. The 
studies create a substantial lien on personnel slpts, which can be 
moxe. valuable than the money itself in times of freezes on hiring or 
downgrading. Agency leadership may be conceptually supportive of the 
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studies and* honored* by Congressional confidence in NIE, but in practical 
terms, the studies are viewed officially as external to the agency's 
mission. This view has conseqxltences affecting studies which are 
.legislatively rec^uired to tneet deadlines and are on a fixed income. 

One example i^ personnel. How many slots and at what levels 
have to {^ negotiated with agency leadership, which understandably 
would prerer to lose as few slots as possible to frozen money studies i 
Negotiating the vocational education study personnel configuration 
* has taken over a ye^r before the study finally received what we regard 
as authorization. to recruit enough of the right mix of positions to 
carry out the study. Hiring persoYinel for these positions is another 
challenge. The Civil Service procedures require (a) review of the 
^ position descriptions prepared by project leadership for consistency 
^^th the agency-established organization chart, (b-) establishing a 
gtade level consistent .with Civil Service regulations,', (c) posting 
these positions, (d) review of *the applications by a special panel 
to select highly qualified^ qualified and non--qualified candidates, 
^e) interviewing candidates and preparing selection reco.mmendatipns, 
(f) approval of the recommendations by three echelons, (g) negotiation 
of salary and other terms of employment with the candidate and 
(h) arranging for initiation of employpent. All steps except (g) are 
the responsibilities of different functionaries in the , Personnel Office, 

^ each of whom has many other actions which can take priority unless 
there is an override. In the case of the Vocational 'Education study, 
the Associate Director of the Education and Work Groiu) placed Study 
p^sonnel actions at the head of all other EWG requests except con- 
tract renewals for already hired personnel who would .otherwise lose 
their jobs. *As a result, other EWG staff have been carrying respon- 

visibility far the work ^of several unfilled vacancies for two years or 
longer. Even if all goes well^ recruitment requires about ten months 
between approval of a staffing chart and arrival of the staff member. 
With only 12 months to prepare a study plan, having 'to spend the first 
six months or longer trying to assemble a t^am can be a Hobson's 
choice between going ahead with pl^s the team may later wish^ to undo 
or delaying work in the hope the team can do twelve months' work in 
six months when it finally arrives. . « , 

s 

As a second example, awarding a contract requires (a) approval of 
review panel appointments, (b) preparing the scope of work, <c)' approval 
of the scope of work by the review panel or revisions, (d) approval, 
by the Contracts Office or revisions, (e) announcement in Commerce ^ 
Business Daily, (f) ho lading a bidder *s conference and mailing out the 
minutes of the meeting, (g) receiving the proposals, (h) review of 
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the proposals, (i) negotiations with bidders in the eligible,",, 
range, (j-) receipt of best and final .offers, <k) review" of best 

1 and final offer's, (1) preparation of the ^tnemo of- re'conmendationj 
(m)..,final negotiations and (n)>contract.a^ard. ,bnly the pace of 

■ activities b, h, k, and 1 can be influenced by project staff. -For 
all.the other steps, the Congressionally mandated studies have to ' 
netotiate a place in line among many other contract actions trying 
to get through the narrow? 'contracts funnel. Long as four years 
may seem, the full year th^t spme contract actions have taken in 
the-past would be too slow to pe'rmit meeting .Congressional deadlines. 
In some instances, study directors have ha*> t>o choose between the 
hope of later legitimation of work informally initiated or having 
to request an extension of time from Congress. . ^ 

•As a third example, the Congressionally mandated studies and 
also funds transferred from other agencies have been declared out- 
- side of the agency's mission. A charge is therefore levied on the 
gross budget to cover telephone charges, equipment, space rental, 
and other .overhead items such as percentage of all costs of contracts 
^and personnel staff. . The direct persorthel costs of projeqt staff, 
their travel and consultants also are takep directly out of project ■ 
funds, although the Staff come from regular agency slots. This 
reduces the $1 million annual appropriation for the vocational 
education study to $650,000 annually available for all data collection 
and analysis for all four Congressionally mandated sub-studies. 
Over 20 projects are required in the plan, yet $650,000 is about 
the cost of one national evaluation. 

^ _ . There are strongly held views on the reasonableness of these 
situations. With backlogs in personnel and contracts actions, 
often resulting from freezes, reorganizations or Civil Service reviews - 
beyond agency control, it is hard to justify penalizing ot*er agency " 
projects because one study has its source in specific Congressional 
, interest.^ Nor is there justification for starving the agency as a- 
whole for staff^^and expenses' money because one favored study needs ■ 
S&E, too. But It does create tension when a project is "taxed" for 
support services which it then Receives only slowly and painfully 
because no new. staff ' are hired in personnel or contracts to provide » • - 
^ the support setvices. On ^ small scale, these inconveniences' can be handled 
on a caser-by-case basis. Oh a T.arger- scale,. viewing Congressionally 
mandated studies as external to an agency's mission suggests some 
low upper -limits on agency capacity'to manage Congressionally required 
studies .while retaining its identity as part of the Executive Branch. 

■ . ' "i ■ • . 
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* One study manager illustrated this tensionr . ^ , ' 

^ "I desperately needed someone -to do some multivariate ' ^ 
analyses. The Associate Director in July isr76 pro- 
mised to get us a person for a year.. We had a good 
•v^ • list of six or seven candidates. Negotiations to get 

authorization for this one person dragged on through 
October-and November and the candidate kept dribbling 
away. In January we finally got authorization to hire 
this one person. We had to hire the last person on 
th^ list and he decided at' the last minute to .take a 
'job elsewhere. It was now February. We finally hii;ed 
a person in March and we had expected to haVe someone' 
in November. 'I wouldn't direct, a Congressionally 
mandated study again unless I could set the personnel- 
authorization up right from the beginning/' 

A consensus among mandated study directors seems to be that 
in order to permit agencies to operate.more liTce private research 
^ . firms, .which is needed if the studies are to go well, the legislation 

authorizxn^ the studies should also authorize (l) additional positions 
above the agency's present ceiling, (2) funds to cover th4 cost of 
' -f^f" and expenses as well^ as the costs of contracted, data colUction, - 
. UJ use of non-standard procedures for hiring gnd recruiting per- ' 

sonnel and (4) use of non-standard ■{)rotedures for awarding contracts ^ 
and grants. ' • ^ • 

Rara Avi?, Different Staff: Almost any evaluation team needs to find 
a path among various groups. who wish to protect the program being" 
evaluated and those who want some muckraking, b4tween responsiveness 
to. the needs of various clients and stakeholders and concern for the 
credibility of the study. The skills required" to do so in a Con- 
. ^ gressionally mandated study are of an order of magnitude different ' 
from those of usual evaluations, even national' evaluations . Communi- ' 
, cation between a program staff member of the Executive Branch and ' 

Congress, which is most unusual and indeed usually is fox^dden, now = 
becomes central. A study director who can not communi'c^^com'fortably . 
freely and coUegially with Congressional clients has lost what may 
. be the greatest advantage of the new Congressionally mandated'studies . 
Persons who combine research experience, the bureaucratic and 
managerial skills required to complete a program of studies,' and the 
•x^ superb communication skills needed are difficult to locate. We 

hope we have put- together a team for the vocational education study • 
whi^h can come through, yet we are aware that the Compensatory 
Education study may have been remarkably blessed in having a full- 
time, highly experience'd Congressional liaison and in the communication 
^ , skills of Its Director. i 

ERIC ^ , 12 . . , ^ ' . ' 



• The requiremeht for political sensitivity extends to other ^taff / 

• Many educational researchers seem bate ly aware of the difference between 

* appropriations Sand^ authorizations, ^do not kMw wH^n .ife^jortant pieces 

of legislation in their field are scheduled%or re-authorization,' and * 
-^find it djrfficult to discuss a question such as "What research^wouid* . 
be useful for a Congressional comfiiitlfee considering r^autj^|HB||M|^^^^ 
of any pi6ce of lej|^islation closest to your interest?" 'f^^^^K^^* * 
between Cohgressional-'concei^s and 'the perapectives even^Rjj^luators / 
in Wa^ington agenc^s is Mten leagues-wide. 

In addition to cominunication skills and sensitivi)^ tp policy * ' 
questions, the staff need, to be p'^rceived as honest b^tfokersjv persons 
informed about the concerns of various interest groups, who are- neither 
advocates' (which would limit their credibility outside of the program * •* 
being, s tudiefl) Adversaries (wHich wpuld increase the difficulties 
in conducting tlje study and limit their credibility within the program 

■ being studied) ^ "Who is on your sta^f" is .often the first question 
we are asked: special interest groups want <;o" see many of their own 
leading the study while public interest group^wafit to feel assured 
the study is not a creature of the spetial interests .hence,- the ' ' . 

^ value of tl^e^, honest broker. , . , - • . 

Centipede Agonjstes r 'The evaluation Btudie^ ma^iciated by Congress 
may not be intended as parts of a whole, bufc for logi's^ic^as well 
as conceptual reasons, they cfm not work at cross-purpose^s. The 
1976 vocational, education evaluation rejpirements include (a) the NIE 
studies, which will investigate f^ur distinct subqu4stions , (b)' the 
national' study mandated by Cdngr^ss of sex-fairness in vocational : 
education, wjfiich is being conducted by, the USOE Office of Evaluation 
and Dissemination, (c) the National Vocational Education i)ata System 
which is to provide information on enrollment^, completions, staff, 
facilities, curricula icad cost^ and which is operated by the National 
Center iox Educational Statistics, a% required Congress, an' office ' ' 
reporting to th|^ Assistant Secretary , for Education, Xd) /the in-depth 
evaluations by USOE of ^ach*.of the state-level vocational education 
-program^, ^which are to b4 conducted atv a rate of teri annually, 
(e) the *in-depth evaluations by each state of the -vocational educatioR- ^ 
programs and activities -within that state, to be conducted a.t*a#rate ' 
asswing that ey.ery program will bV reviewed at least once during ' i 
the fjve year period, (f) an annual follow-up study of program \ ' * 
completers and tipn-complete;rs to be conducted on a sampling basis by 
ev;ery'state for eve^ yeai||(g) the evaluation of the federal adminis- ^ 
trktioh "bf vocational Education to be conducted by the National ^ - . 
Advisory Council ofi Voc^tibnal Education and (h) the evaluation 
stuties,j:o be ^coriducted by the^ew National Center for Research in ' 
Vaca,tiot4l Education, funded £0Kfive years at. $5 million annually. 
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^USOE 'cjentr^l Office of Evaluation usually c^d^ucts qj^" 
s of vocational educat^ion; the Bureau r.esponsible 
the^ Vocational Education Ac't , has -authorization 
ndments tt) expend'all *of "theJFederal share of the 
on contracted activities, ;^hich. typically ^re more . , 
I than grants; and the GAO is *c(insidering undertaking 



its owfi evaluation of the impleme'ntation of th^* 1976 Amendments 



This poses. some challenge for .coordinating dSta collective and . 
• integrating -findings. Suppose for One 6f the NIE studies, it is 
important to know the rate of change fey gender in enrollments in 
occupational and non-oqcupational^ vocational education programs. One 
approach is fo^ NIE to design a survey which will answer this question . 
directly, To^ -avoid duplication of effort and unnecessary demand^, on 
the schools, we will nee^d to find out (a) if the N\^DS system will have 
reliable data -by ng later than January' 1981 ori a natigjially represen- ~ 
tative basis with the new uniform definitions available for 1977, 1978, 
1979 and 1980, (b) whether the sex fairness study will include a large 
enough sample of enrollifaents oyer four years to permit generalizations 
about occupat^ional and non-occupational enrollments, (c) if data on 
rate of change will be available on large enough and representative 
enough s amp lei^ with uniform definitions from the USOE evaluations o£ 
the ten states annually or from the state evaluations' of the local 
programs, or (d) if there is another Federally funded evaluation 'from 
which th6 data would be available. 

Multiply this bne example by scores of data points, and the 
coordination challenge becomes substantial. The agencies have • 
appointed liaisons to each other ^s studies and we are working with • ^ 
the Chief State School Officers Committee on Educational Information 
^ Systems and with as many state coordinators of data collection as ' 
possible. Sharing draft RFPs and serving on each other's review - 
panels helps • There may be some point of no retir^t]), however , .where 
taken as a*i7hole within legislation, there may be enough pf even such 
a good- thing as evaluation, . ' , . 

While coordinating data collection dominates discussions now, 
in three years the issue may be integrating findings,. For the 
Congressionally jnand^ted study, synthesizing the interpretations, of 
over 20 projects will be* a difficult tas-k. There is no provision . 
for ,anyone to integrate data, interpretations and recommendations of 
the NIE= stpdies with results of the.many other evaluations. * 

The two other Congressionally mandated studies did not attempt 
an integration across non-NIE evaluations. At least one instance in^ 
the compensatory education hearings illustrates the confusion possible 
when studies reach different conclusions* This instance dealt with 
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the liotly debated question of the proportion an<f numbers of children 

eligible for assistance if low income, poor achievement/* or only 

the intersect of these was the criterion (Hearings, 1^78, pp/ 162-165), 

' Some provision seems needed to assure a thoughtful, well-informed, expert 
-analysis of conflict and congruence in study fin^dings;' probably a tl>ird/ A 
party not directly responsible for conducting any of the studies such 
as the Concessional Budget Office or the' Congressional Research Service, 

* should be mandated (and funded) to undertake such analysis in time for the 
reauthorization hearings * 

Holding A Steady Course in Unsteady , Waters : Evaluation is a slow ,enter~ 
prise. The pace of design, measurement, data collection a'nd analysis is. 
deliberate and a studyyCSh not easily be directed toward a new port in 
mid-course. Yet natioftal-level forces require considerable tacking^^as 
winds, shift and waters flow. ) * ^ 

. / - ' - — 

4s one example, court-ordered bussing to achieve desegregation can 
change clas^ and staff composition from one year tp the next and even 
within years, affecting aMlity to disentangle effects of curricula 
within classes from other influences, some of^ whichSpay augment the 
curriculum's effects and some of which may be disruptive". In Follow 
Through, the almost- continued shifting of 'principals, staff and students 
to achieve desegregation seems likely to have been accountable for some 
of the site-to-site variation, yet this was not systematically examined 
m the Follow trough evaluation. * ' 



Turning to the vocational education* study, the decision to centralize 
the functions of the USOE regional offices may affect very seriously, 
capability to perform the Federal administrative functions required by 
the 1976. Amendments. This reorggfiization already has coiflcided with 
the need to review and provide technical assistance to the more than 50 
states in their five year plans, a demand that is less than perfectly 
met by regional staffs in transition and in. some panics.. Effects on " 
timelines and throughness of the reviews may be expecKed. The result, 
from a research perspective, may confo^ind a possibly excellent planning 
process ia the 1976 Amendments with a major reorganization. -^The over 
12 months' delay iir appointing a Deputy Commissioner fof the Bureau 
of Occupational and Adult Education may also have profound influences 
on implementation of the 1976 Amendments in ^ays'that confound -this ' 
hiatus in 'leadership with the new provisions to improve planning and 
administration. The .decisions that threatened to eliminate appropriation 
requests in FY 1979' for Consumer and Homemaking Education.may have 
influenced how Fedei-al, 'st«te*and local C&HE staff spent .their energy 
in the past few months, and alsp have had consequences throughout the. 
C&HE system in allocation of time in responding to constituency concerns 
versus time for new leadership. 
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The challenges these secular changes pose to evalu*ation of the 
'Legis.lative Word is not antiseptic, pcotection of a program from the real 
world in which it is being studied. Th6 challenge is being sensitive 
enough,- being able to respond' fast •'enough to recognize and document 
these influences, and to negotiat^ changes in the study wisely enough 
that the results accurately yef le^t th6 pniltiple influences on 
implementation of the yocdtional education programs. 

Heisenberg, the Witdg Doctor. " A^Congresaionally mandated study by its " * 
very existence may stimulate program change. A GAO audit or a mandated 
?Q"?Le-%rHL^K ^"^'^ an -indicator of Congressional intent'' 
CO take action based on the study fin^ii^s more so than most evaluations. 

. The initial response of t!he vo>:ai:ibn^ education 'field was negative 
toward having a study at all, 'toward NIE as/the' responsible agency and 
toward Its assignment to the Education and Work Group. It was feared 
that the NIE was biased against vocational ^uca'tion (because NIE had- 
funded the Wilts' study wh'ich was mo'sf unpopular among vocational educators 
1975J and was too ignorant of the history of the ptogram to -design 
and conduct an adequate study (because nosone in EWG had been a 
vocational educatariand because.NIE was incorrectly psrceived as not 
having funded vocational education reaearch^Stevenson, 1974 v. StumpJ 
1975^. Tii^resuit, it was predicted, 'would be a- waste of millions 
of dollars on useless, biased, inadeqifate data that more knowledgable 
people would then have to go to the-. trouble of refuting. Vocational 
educators, fought,- furiously, to delete ttiS NIE study from the 1976 Act. 

■ Over the past months , vocational educators seem to liave focussed 
on the i^eed'to get the .job ..done with regard to implementing the 1976 
Amendments,' using the resources of the' profession to carry out th^ ^ 
many parts of this new legislatio^.as, rapidly and adequately as possible. - 
. The evaluations may be 'regarded still &s time-bombs that may need 
politicaT^action to defuse,^ but the fact that there will be so^^many 
^dependent reports on the 1976. Act seems now to be part 'of the motor 
' driving^ the changea themselves. "The goblins aare going to get you if ' , 
you don t watch out may inspire .ascertain amount of wat'ching out to , 
see. If the ^blans kxe gaining, but, it may also inspire additional 
effort to assure the programs generally could meet the standards of 
excellence of a reasonable evaluation. , To ,the extent that the 
Congressionaily 'mandated studies are part of this stimulus, their conduct 
will confound other provisiorts in the 1976 Amendments to achieve' 
compliance with the intenJ: of Congress with the process of being evaluated. 
While It isn't clear whether the new Congressionally mandated^tudies 
generate mare eneogy directed toward achieving the intent of legislation 
or more energy directed to. evading evaluations, my impression leans 
toward the former. ' As a consequence, the,' study will have to T>e self- 
conscious in,doeumenting the influence o£ its existence on the variables 
bein^ studied, in addition 'fe^tudying 1:he effects of other provision's. 
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^t?hL°r? ^:ongre38i6naUy tnandated |ftudies bring 

b^Sp^ ; f • f ^ i"^' '^^"^ directed^to'one purpose a'nPwith ,a " 

provide? "^^^^ ""."^^ administration - 

.^providej/The rewards as well- as the risks are clWrer for the 

researchers.^ ?he team is a, one-time group ^hich need' not expend ' 

energies on protecting its turf in the annual budget cycle Because V 

r of the e^caitetaent of 4 Cong^^ssiortai study, and the great support of 

«?inL: efgr^e^"' indicating V':rth:r:t d es 

are intended .fdlr use, a goed team can be attracted. Working on'a 

dif?!' „ '^''■"'"''"""^'•"^ ^project :likely. to. make a n'afional 

L aXr::ucraT T%h"'"1^'^"'='^" "^"^^ -^^-^"^ admiSstration 
^?ft!^ • »^sueUp-attem. an, evaluation project 

use or,h:.nn-'^'- ^---y-l'^iisparate studies at'one time. The 
-s^Uer r^f"-"' ^^■"^""^in, and, becausf the studies are of ten 
n^Ji!J? cohesive -team is not formed. The loyalty of the project 
officers more tdvrard the bureaucratic unit, which spend* consiSeraMe 
.^nnual, energy on annual planning in irder to survive In t^e new 

-ii'I'drit [f'^i'f CongrLHnd'to 

study .Itself. It IS also more rewarding professionall? to olan 
an evaluation program of related studies which together can l^swer 
L^g^^t^^ ^° on!-contract-|!:^ime/ 

' rewar5^%lreadreaLed\v°'°''r^'°l Education Study and the professional - 
rewaras already gamed by members of the study team adds to morale on 
the new studies. .There is belief in the potential of this fo^^n/ ' 
enthusiasm for being ^6ng the-first by whom the new is trifd Se 

Sir eel' ?rb ' often ppwerful n"io"al agf;cy^truc- 

ture seetas-to be of advantage to study conceptualization in^™1 • 

CongressioMl conmittees and for the tvIl^Lorf '''f/^-lts for the 
.Being in clo=« contact than usual »iC i^ ° ' " 

.■■ ctj e"tL"::"o" Jnfo^f f -formation- cafhejp ^o'ngrelaional 

h<. fo«! °ecome more informed consume^of evaluations;. There mav 

' ■ undfr^eLrT'^^'f °' """^^ evaluations can accomplish and less * 

. underrestimation of the certainty of conclusions. . 

- opportuniS%r!irr'''* "^'^ -Congressional, staff and th'e ' ; 

^Et^Fr^ -af iv-f io"er=-?n-r 

xn both direcrt and more subtle ways all aspects of the studies! Se 



evaluators develop a third ear fojj.what might or might noX be useful 
to Congress, how the questions can be phrased, ^and how tHe finding 
comnrunicated. . As the Hearings (1978) suggest, the evaluators are 
^^bettet prepared to- communicate with Congress and^ to* handle qi^estions 
such as whether litleJL/is' serving more non-T)oor than poor i^il^ren. 
While the javaluatioris can not be ^directed ftom day-to-day. by Congress 
without* losing their objectivity, through co^nication and negotiation 
sensitivities develop that influence the credibility of the findings, 
a factor that may be the mostxinfluential of 111 in .determining • 
utilization. ^ 

' In Conclusion s Loojking. at experience* to date,, it is perhaps too soon 
to say whetlTfer the feature,s that contributed to the positive re^e^tion 
of 'the Compensatory Education StudJ reports are repeatable, or even to 
be sure which features were new and essential. Perhaps it was .due to a 
one time happy coincidence of g remarkably skilled Project Director, 
Paul^Hill; ^'outstanding team; an -appropriately mandated study 
funded with enough money for a long enough time to be achievable; a^d 
a period within the Executive branch of something like bianign neglect. 
The Vocational Education Study has been managed differently, and it is 
not clear if sjtudy features are reproducible in whole or in part 
(e.g., the negotiated design aipects} under a ^ider variety of cir- 
cumstances, or how many constellations of ^pproache^wLthin the 
boundaries^ of the mandated 'study model may be successful. 

For an approach S9 widely used, there seems ta be remarkably 
little research on Cong:fessionally mandated studies, either new or old, 
and veryi little feedback from the s.tudy teams to help Congress write 
mandated studies likely to achieve what Congress has most wanted. " 

acking such analyses, there. are reasons for caution in endorsing 
all 'or some of the features of the new Congres^ionajly mandated studies 
as a better route ^0 evaluation utilization*. 

At least this is the*cautious per^pe<ftive of an evaluation * 
researcher. As a member of the study team, I. am heartily Enthusiastic 
about our vocatiofial education study and about its progress to date. - 
I am also looking forw ard to at least two other meetings with AERA# a 
few years from today^ fan'dT tnej ^c ationai education community Thefirst 
might present the findings of the 'study and how t;hey have been used in 
the -1982 re-authorization hearings. And the second would describe how 
the vocational education study 'procedures have themselves expanded the 
development of a new species of evaluations; , Congress ionally mandated 
studies that are both useful and used. 
- H , . 

See Weiss, 1970, however, for an ^exceptionally illuminating 
case study. of Federal Student loan programs.) 



